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Abstract 

Spectral clustering methods have been shown to be effective for innage segmentation. Unfortunately, the presence of 
Image noise as well as textural characteristics can have a significant negative effect on the segmentation performance. To 
accommodate for Image noise and textural characteristics, this study Introduces the concept of sub-graph affinity, where 
each node In the primary graph is modeled as a sub-graph characterizing the neighborhood surrounding the node. The 
statistical sub-graph affinity matrix Is then constructed based on the statistical relationships between sub-graphs of 
connected nodes In the primary graph, thus counteracting the uncertainty associated with the image noise and textural 
characteristics by utilizing more information than traditional spectral clustering methods. Experiments using both synthetic 
and natural Images under various levels of noise contamination demonstrate that the proposed approach can achieve 
Improved segmentation performance when compared to existing spectral clustering methods. 
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Introduction 

Spectral clustering is a powerful tool for machine learning 
designed to cluster similar data, making image segmentation a 
popular application. Zhang et al. [1] segment plant types from 
SAR images using a ensemble-learning and spectral clustering. Lui 
and Jia [2] apply texture characteristics and Markov random fields 
with spectral clustering ensembles to segment land types from SAR 
data. Sundaram and Keutzer [3] apphed spectral clustering with 
GPU parallelization to segment people and objects from video 
sequences. As a medical application, Ge et al. [4] apply spectral 
clustering to segment the corpus caUosum white matter fibers 
connecting the left and right brain hemispheres in diffusion tensor 
imaging (DTI). Carcassoni and Hancock have developed tech- 
niques that improve spectral clustering applied to correspondence 
matching improving the robustness of matching similar structures, 
demonstrated on object clustering within images [5,6]. 

For each data point, or each pixel in the domain of image 
segmentation, a similarity matrix or, more generally, an affinity 
matrix, W is constructed. Each element in the matrix, W^^, 
represents how similar pixel u is to pixel v. The similarity matrix is 
transformed into an eigenvector domain, allowing the derived 
eigenvectors to provide a means to identify the most significant 
features within an image. Ideally, each salient image region 
transforms into a cluster of data points in the eigenvector domain. 
Using a clustering algorithm such as k-means or hierarchically 
clustering, the resulting clusters are identified and labeled. As long 
as there are a sufficient number of good measurements, spectral 
clustering can be very effective at segmenting an image. 

Two important challenges faced when applying spectral 
clustering to achieve image segmentation is dealing with the 



presence of i) bad measurements and/or noise-corrupted mea- 
surements, and ii) textural characteristics and variations within 
image regions. Hong et al. [7] encounter this situation in 
underwater sonar images. In such situations, the pixels within 
the image may be misclassified using existing spectral clustering 
approaches given their heavy reliance on the affinity between 
connected nodes in constructing the affinity weighted graph; the 
pixel intensities can differ significantly due to noise contamination 
and textural characteristics and variations. While Hong et al. 
improve spectral clustering through morphological wavelet trans- 
formations for sonar images, they require additional post- 
processing to the spectral clustering framework and do not 
account for any textural characteristics within the image. 

Expanding on pixel-to-pixel nodal affinity, Pardo [8] touches on 
the idea of using region statistics to better classify the edges of 
regions in noisy images. For each region, Pardo calculates the 
standard deviation and mean, allowing regions with similar 
statistics to be combined into a single region. Comparing two 
region statistics assumes that each region has uniform intensity, 
which does not account for textures or region structures that can 
be detected through a sub-graph method. Therefore, a means to 
provide more robust image segmentation under heavy noise 
contamination and in situations where regions exhibit textural 
patterns and variations is desired. 

The main contribution of this study is the introduction of a 
robust spectral clustering approach to segment images under the 
presence of heavy noise contamination as well as the presence of 
textural patterns and variations in image regions. Instead of 
relying on the node affinity between connected nodes in the 
construction of the affinity weighted graph, the proposed approach 
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utilizes the concept of statistical sub-graph affinity between 
connected nodes. By taking into consideration the spatial-intensity 
relationships within sub-graphs, the assumption is that the 
statistical affinity between sub-graphs is less sensitive to noise, 
which is random in nature and does not typically exhibit such 
structured relationships, and better characterizes the underlying 
textural characteristics. 

By themselves sub-graphs are not unique; Luo et al. [9] also 
utilize sub-graphs, they improve robustness by clustering graphs 
relating to the structure of objects, applied during the clustering 
step of spectral clustering. Instead our method applies sub-graphs 
to extend the affinity metric to include a comparison of the 
neighbouring regions; the sub-graph affinity utilizes region 
statistics before the clustering step so that similar regions, instead 
of similar nodes, can be clustered more robustly Our work focuses 
on extending the nodal affinity step of spectral clustering to a 
statistical sub-graph affmity. 

The rest of this paper is organized as follows. In Methods, we 
explain the methodology of the proposed statistical sub-graph 
affinity modeling approach. In Results, experimental results using 
both synthetic and natural images under various levels of noise 
contamination are presented and discussed. Finally, conclusions 
are drawn and potential future work are discussed in the 
remaining two sections. 

Methods 

In this section, the methodology behind the proposed statistical 
sub-graph affinity model for spectral clustering is discussed in 
detail. First, the traditional node affinity model for spectral 
clustering is described for context. Second, the proposed statistical 
sub-graph affinity model, which is designed to better account for 
noise contamination and textural characteristics and variations 
within the image being segmented, is presented. 



Node Affinity Model for Spectral Clustering 

In the node affinity model for spectral clustering, similarity 
metrics measure the difference between two nodes and are 
typically scaled between zero (dissimilar) and one (similar). Given 
an image / of size m x «, a given pixel u has a node measurement 
Xu that direcdy corresponds to a grey-scale intensity, 

Xu=Iij- (1) 

Each edge, w^,, in the affmity graph represents the node affinity, 
or similarity between two node measurements, x„ and x,,. The 
affinity graph can also be represented as a weighted adjacency 
matrix W, where the elements are equal to Wuv In [10,11], the 
weight between two connected nodes u and v is computed based 
on their node affmity. 



= exp 



-(A-„-x,,y 

2ff2 



(2) 



Using the affinity graph with weights defined in Eq. (2), many 
authors have contributed variations that produce different 
clustering results depending upon circumstances [12-22]. Perona 
and Freeman [12] calculated the eigenvectors of W and applied 
thresholding to the principal eigenvector in order to separate two 
classes of data. 

Instead of working with W directly, Shi and Malik [13] 
computed the unnormalized Laplacian, 



L = D- W, 
where Z) is a diagonal matrix defined as 



(3) 



5 




Wuv = exp 



^^ 2(t2 

(a) node affinity 



exp • 



2(t2 



(b) node affinity witli 
neigfiborliood statistics 



2a2 

(c) sub-graph affinity 



Figure 1. Comparison of node affinity to sub-graph affinity. This figure illustrates a pixel-to-pixel rectangular sub-graph specific to 
implementation for an image and the corresponding edge weight equations, (a) For node affinity, each node, .y„, (red) is compared to a neighbouring 
node, .T,, (green), for all nodes in the local neighborhood, JSf. (b) If neighborhood statistics, such as a mean or weighted average, were applied as 
preprocessing to the data or image, the resulting similarity metric would contain regional statistics and would be more robust to noise with the cost 
of losing fine details such as edges and textures, (c) Rather than calculating the similarity of two regions from a single statistic, as in (b), sub-graph 
affinity calculates the similarity of two regions by calculating the similarity of each corresponding element within the regions, improving robustness 
to noise while preserving fine details of the region. Using sub-graph affinity, two regions with the same average intensity, but with differing textures, 
can be classified as two different classes. 
doi:1 0.1 371/journal.pone.0082722.g001 
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(4) 



Shi and Malik applied thresholding to classify data points into 
classes based on multiple principal eigenvectors. The matrix L 
associates the number of connected components in the graph, kcc, 
with the multiplicity of the 0 eigenvalue of L. This allows the 
number of connected components for a particular system to be 
found. This approach was subsequendy extended to incorporate 
contour and texture information [14]. 

Scott and Longuet-Higgins [15] extended the thresholding 
methods with more generalized clustering techniques to improve 
clustering results. Ng et al. [16] introduced normalized Laplacian 
matrices, 



L,y„, = D-'^I^WD-^l\ 



(5) 



and clustered the principal eigenvectors of L using the k-means 
clustering algorithm. Variations of this method also include 
defining an asymmetric normalized Laplacian as 



L = D-^W. 



(6) 



which has properties similar to that of a random walk. An in-depth 
discussion of the various connected-graphs and affmity matrices is 
provided by Chung and Mohar [17-19]. 

One of the key hmitations with the node affinity model for 
spectral clustering is that, by only taking into account the 
intermediate similarity between two node measurements, the 
information available for assessing affinity is relatively limited and 
as such can be sensitive to the presence of noise. Furthermore, by 
only taking the individual node measurements into account, the 
node affinity model provides a relatively poor characterization of 
the textural characteristics within the image, thus limiting the 
ability to account for situations where the image is characterized 
by highly textured segmentations as well as textural variations. 
Therefore, the main motivation behind the proposed statistical 
sub-graph affinity model is to counteract the uncertainty 
associated with the image noise and textural characteristics by 
utilizing more information than the node affmity model. 

Statistical Sub-graph Affinity Model 

The proposed model attempts to improve the robustness of 
spectral clustering for situations characterized by heavy noise 
contamination by replacing the use of node affinity described in 
Eq. (2) with the concept of statistical sub-graph affinity. 

In order to create the affmity matrix, the connectivity of each 
node to every other node must be represented in a connectivity 



K 7x7 



Vk.P^ 3x3 




v«.J'. 5x5 



K. 13x13 




neighborhood 



Figure 2. Effects of graph kernel size and sub-graph Icernel size on neighbourhoods and algorithm complexity. The effects of 
increasing the size of the graph kernel, K, and increasing the size of the sub-graph kernel, Vu, are illustrated using a two-dimensional example above. 
As the size of K increases, from top to bottom, the size of the neighbourhood (solid dark blue) surrounding .y„ (dark red) increases. As the size of Vu 
increases, from left to right, the size of the sub-graphs (light red and light green) increases. Increasing the size of decreases the sparsity of l/Kand 
increasing the size of Vu increases the computational complexity of calculating ir„,., but only moderately. 
doi:1 0.1 371/journal.pone.0082722.g002 
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graph, A/". When dealing with large images, memory and 
computer resources need to be conserved; to do so, N is implicitiy 
constructed from the graph kernel K.. The kernel weights, elements 
of K,, represent the degree of connectivity between a node and its 
neighbor. 

The novelty of the proposed method is the introduction of a 
statistical sub-graph affmity model to replace the node aflinity 
model; instead of comparing jc„ to Xy for all v in the neighborhood 
of M, the sub-graph Vu is compared against each sub-graph P,,, 
which is illustrated in Fig. 1. 

A statistical mathematical description of the process is required 
to understand how sub-graphs can improve classification accuracy. 
Given a node u belonging to class Ck, the nodal features, x„, can 
be modeled as the random process, X„, affected by independent 
and identically distributed (i.i.d.) noise, N: 



weights and the statistical sub-graph affmity, can be defined as 



(7) 



where /ij. is the mean of all nodal features belonging to class Ck- 
The effect of noise is reduced by incorporating neighborhood 
information contained within a sub-graph, Vu, which stores the 
connectivity of node u to its neighbors. When using node affinity, 
the probability that that two nodes, u and v, belong to the same 
class, P(ueCk and veCk), denoted as Pu,v,node, is the probability 
that the difference between X„ and is equal to zero, 



Pu, v,node — PiXu — 0 1 Ck ) • 



(8) 



Since X„ and X, are i.i.d, then Pu,v,iiode can be determined using 
the Laplace distribution. 



(9) 



where b is the scahng factor associated with A'. Compared to node 
affinity, sub-graph affinity suppresses noise. The resulting sub- 
graph affinity probability, Psuh, is equal to. 



Pu,y.sub = n P(X„,, - X,j = 0\Ck) 



-P„,v,.™ft= H— — exp{- 



(10) 



(11) 



where / is the index of each element in Vu and corresponding 
element in Vr- The more similar values in each sub-graph, the 
smaller x„_,- — becomes, increasing the probability oi P{ueCk 
and veC/t); the local neighborhood information suppresses the 
noise. 

Furthermore, the use of sub-graphs allows the spatial-intensity 
relationships between nodes within a sub-graph to be preserved. 
This intra-sub-graph spatial-intensity relationship preservation is 
important as it allows for improved modeling and representation 
of textural characteristics within the image. As such, the proposed 
statistical sub-graph affinity model accounts for uncertainties due 
not only to noise contamination but also to textural characteristics 
and variations within the image. 

Given the proposed statistical sub-graph affmity model, one can 
now incorporate this model into the spectral clustering framework 
by extending upon the weighting metric introduced in Eq. (2). The 
density of W depends primarily on the complexity of A/" and is 
independent ofVu- The values of W, which depend on the kernel 



n 



exp 



2(72 



(12) 



for nodes u and v. The weights associated with JCu and Vu are also 
introduced to allow spatial weighting if so desired. 

Fig. 2 helps illustrate how /C„ and Vu affect the size of the 
neighbourhood and the computational complexity of Eq. (12). 
Increasing the size of K, increases the size of the neighborhood, 
and consequently, the number of x„ and x„ similarly calculations, 
which results in decreasing the sparsity of W. Increasing the size of 
Vu has no effect on the neighborhood size and, consequendy, does 
not effect the density of W. Rather increasing the size of Vu 
increases the computational complexity of calculating Wuv, but 
only moderately, as die difference of A'„ and x^, which can be 
cached after initial calculation. The size of has a greater impact 
on the overall computational complexity than the size oiVu- 

Kernel selection allows the algorithm to become rotationaUy 
invariant, by selecting radially symmetric kernels. The kernels can 
also encode distance information so that the sub-graph weights or 
connectivity can be a function of spatial pixel distances, i.e., the 
intensity difference of corresponding pixels can be weighted based 



Lexicographically unwrap image 
Into a vector 



for each i.j € / 
do ,L-u ^ /,j 



Define graph kernel and spatially 
weighted sub-graph kernel 



Calculate edge weights 

for II ^ 0 to LrNC.TH(J-! and v <-0 to LE.NGTH(j) 
do ttu,. = >Cu_i. 



Apply traditional spectral clustering 
using edge weights 

for It ^ 0 to LENGTH(x) 

do D.„ = E,. 

[V^A] 4- EIGS(Lsym.LENGTH(C)) 
for A- *- 0 to I.ENGTH(C) 

do V(t,:)<-V(t,:)/||F(».:)||^ 

jju t- KMEANS{V. LENGTH(C)} 

Lexicographically wrap vector 
Into an image 

for each i.j ! 
do r,j <- 




Figure 3. Overview of spectral clustering algorithm with 
statistical sub-graph affinity. The image is lexicographically unwrap 
into a vector, spatially weighted kernels /C and V,, are constructed, edge 
weigfits are calculated, and traditional spectral clustering tecfiniques 
are applied before the labelled data is wrapped back into a labelled 
image. Although the kernels are illustrated as matrices above, the 
kernels can be constructed as an arbitrary graph. 
doi:10.1371/journal.pone.0082722.g003 



PLOS ONE I www.plosone.org 



4 



December 2013 | Volume 8 | Issue 12 | e82722 



Spectral Clustering Using Sub-Graph Affinity 



on spatial distance from the center of the sub-graph. The sub- 
graph aflBnity model described in Eq. (2) compares the statistical 
spatial-intensity relationships between the sub-graph around u 

with the sub-graph of v. By incorporating the spatial-intensity 
relationships within sub-graphs into the construction of the affinity 
weighted graph, the constructed graph is less sensitive to the 
influence of random noise on the variations of individual pixel 
intensities, as well as better accounts for textural characteristics. As 
a consequence, the use of statistical sub-graph affinity should be 
more robust to noise as well as textural variations. 

Using Eq. (1) tluougli (5), and tlie proposed edge weights, Eq. 
12, the entire algorithm is presented in Fig. 3. The first step is to 
lexicographically unwrap the each pixel (or region), ly, in I into a 
vector Xu. The second step is to define the kernels K, and Vu- The 
third step is to construct the edge weights, Wuv, from a prxel-to- 
pixel (or region-to-region) comparison, — Xv+,-, for all pixels 
(or regions) in K.. The kernel Vu appUes spacial weighting to pixels 
(or regions) surrounding Xj,. The fourth step is to apply any 
existing spectral clustering technique that can use our edge 
weights, Wuv The result of the spectral clustering technique is the 
labelled pixels (or regions), The final step is to wrap the vector 
yu into image pixels (or regions), Ty. 

Results 

To investigate the performance of the proposed approach for 
segmenting images under heavy noise contamination, the 
proposed statistical sub-graph affinity model was incorporated 
into the Ng-Jordan-Weiss spectral clustering method [16]. For 
comparison purposes, the original Ng-Jordan- Weiss [16], Shi 
Malik [13], and Scott Longuet-Higgins [15] methods the node 
affinity model was also tested. Spectral clustering was executed 



using MATLAB and custom built MEX libraries for efficiendy 
constructing symmetric normalized Laplacian adjacency matrix 
for each trial. 

Even though our framework can segment images larger than 
1024x1024 pixels, test image sizes ranged from 51x51 to 
250x250 pixels so that results can be compared to existing 
methods. The test image set included several synthetic images, for 
initial testing, and 20 images from the Berkeley Segmentation 
Dataset [23], Fig. 4, for thorough analysis. The trials evaluated the 
graph kernel size iterating them over the following sizes, 
{ 1 X 1,3 X 3,3 X 3, ... 15 X 15}; the .sub-graphs were iterated 
through the same set of sizes. Both uniform and Gaussian 
weighted radial graph and sub-graph kernels were tested. The 
trials tested each combination of graph and sub-graph kernel sizes. 

The PSNR ranged from infinity to 20dB as images exposed to 
Gaussian, speckle, and salt and pepper noise were segmented. For 
illustrative purposes, rectangular lattices are employed for the sub- 
graph affinity model in the implementation, although other graph 
structures ma)" also he used. Best results were obtained when IC is 
uniform and has the largest size possible and Pj, is Gaussian with a 
relatively small size in comparison to IC. 

Quantitative Assessment 

To quantitatively evaluate the performance of the proposed 
statistical sub-graph affinity model in the presence of noise, the f 1 - 
measure was computed on the segmentation results produced by 
both the node affinity model and the statistical sub-graph affinity 
model: 

2TP 

2 TP + FN + FP' ^^^^ 




Figure 4. Real-world images used for testing. The performances of spectral clustering using node affinity and sub-graph affinity were evaluated 
using the above test images, reproduced from the Berkeley Segmentation Dataset [23]. 
doi:1 0.1 371/journal.pone.0082722.g004 
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Figure 5. Segmentation results show how well the sub-graph affinity model outperforms several node affinity models. For all tests, 
the proposed statistical sub-graph affinity model performs as well as or outperforms the node affinity model. The dotted lines represent the mean f1- 
measure for the corresponding nodal affinity model. The solid lines represent the mean f1 -measure for the statistical sub-graph affinity models 
associated with sub-graph kernel sizes of 3 x3 through 9x9. The proposed sub-graph affinity model outperforms existing nodal affinity models at low 
PSNR. 

doi:1 0.1 371 /journai.pone.0082722.g005 
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The proposed sub-graph affinity model performs as well as existing spectral clustering methods when PSNR is high. When the PSNR is low, the sub-graph affinity model 

outperforms fl-measures for the Ng-Jordan-Weiss, Shi Malik, and Scott Longuet-Higgins nodal affinity models. 

doi:10.1371/journal.pone.0082722.t001 
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Figure 6. Object boundaries resulting from sub-graph a_nity and node a_nity segmentation compared to ground truth. The object 
boundaries resulting from the segmentation process are illustrated in red; the ground truth boundaries are illustrated in green. In the presence of 
noise, spectral clustering using the proposed statistical sub-graph a_nity model produces more robust results than the other tested methods which 
use the node a_nity model. Noise tolerance is most evident in the 'owls' image, where the segmentation results produced by the proposed statistical 
sub-graph a_nity model consists of large homogeneous segments, whereas that produced by the other tested methods using a node a_nity model 
contain many small noisy segments. Furthermore, the proposed statistical sub-graph a_nity model better handles the presence of textural 
characteristics within the images, as evident in the 'tiger' image; the body of the tiger is identi_ed as a single segment by the proposed model. 
doi:1 0.1 371 /journal.pone.0082722.g006 
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where TP, FN, and FP denotes the number of true positives, false 
negatives, and false positives, respectively, in the segmentation 
results. 

Fig. 5 and Table 1 present the fl -measure re.sults of spectral 
clustering with the node affinity model and with the statistical sub- 
graph affinit)' model compared to ground truth provided by the 
Berkeley Segmentation Dataset [23] against existing spectral 
clustering methods. Spectral clustering using the proposed sub- 
graph affinity model achieve similar fl -measures to spectral 
clustering results for existing nodal affinity model. However, in the 
presence of noise, even a 3x3 statistical sub-graph affinity model 
shows immediate improvements over existing methods. Since the 
statistical sub-graph affinity model uses the spatial-intc-nsity 
structural characteristics within sub-graphs, the statistical sub- 
graph affinity model is less sensitive to random noise that does not 
have such characteristics, and better characterizes the underlying 
textural characteristics in an image, thus leading to improved 
image segmentation accuracy. 

Visual Assessment 

Fig. 6 visually shows segmentation results generated with the use 
of the node affinity model and the proposed statistical sub-graph 
affinity model in the presence of noise. It can be observed that the 
proposed model is noticeably more robust to the presence of noise. 
This is most evident in the 'circle-triangle' image, where the 
segmentation results produced by the proposed model consists of 
large homogeneous segments compared to the node affinity model, 
which consists of many small noisy segments. Furthermore, it can 
be observed that the proposed model better handles the presence 
of textural characteristics within the images. This is most evident 
in the 'tiger' image, where the proposed model allows the body of 
the tiger to be identified as a single segment, whereas the node 
affinity model identifies the body as multiple smaller segments. 

Conclusion 

In this study, a robust spectral clustering strategy is introduced 
to better handle situations characterized by heavy noise, as well as 
situations containing textural characteristics and variations. The 
concept of statistical sub-graph affinity was introduced for 
constructing the affinity weighted graph based on tlu- spatial- 
intensity relationships between connected nodes. Experimental 
results obtained from both synthetic and natural images under 
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